[DRAFT] FEAT Add Turkish Conversation Prompt-Injection dataset loader#2173
[DRAFT] FEAT Add Turkish Conversation Prompt-Injection dataset loader#21733nesdeniz wants to merge 1 commit into
Conversation
|
@microsoft-github-policy-service agree |
|
|
||
| class _TurkishConversationPromptInjectionDataset(_RemoteDatasetLoader): | ||
| """ | ||
| Load the Turkish Conversation Prompt-Injection Dataset from Hugging Face. |
There was a problem hiding this comment.
Is there a paper associated with this? How were they collected or curated? Do they actually work on some model?
There was a problem hiding this comment.
Thanks for the questions. There is no associated paper or preprint at this stage. The citation refers to the versioned Hugging Face dataset release itself.
The dataset was created through a paired curation process. I defined ten prompt-injection attack families and prepared 150 attack examples. For every attack, I created a matched legitimate request that remains close in topic, context, and vocabulary but does not attempt to override instructions or cross a trust boundary. These boundary pairs were combined with 450 additional legitimate requests, resulting in 600 benign examples and 150 attacks.
This structure is intentional. A detector should not classify a request as malicious merely because it contains terms such as "system prompt," "admin," or "API key." It must distinguish between legitimate user intent and an actual attempt to override authority or extract protected information. The matched benign examples serve as hard negatives and are intended to support false-positive analysis and reduction.
The attack examples were also executed in AltayDuel, AltaySec's continuously operating AI-vs-AI prompt-injection arena. AltayDuel runs Red and Blue agents across multiple model providers and records the complete conversation transcript, model and provider metadata, and judge verdicts. The attack examples included in this release produced successful prompt-injection outcomes under the arena's documented evaluation criteria.
These results should not be interpreted as universal bypasses against every model or configuration. Prompt-injection success remains dependent on the target model, system prompt, available tools, and surrounding application context.
The dataset repository also includes deterministic pair-preserving splits and validation for schema consistency, duplicate rows, pair integrity, split leakage, attack-family balance, and common sensitive-data patterns.
2 participants
Description
Closes #2171.
This PR adds a first-class PyRIT loader for the public Turkish Conversation Prompt-Injection Dataset (CC BY 4.0). The dataset contains 750 curated Turkish examples across 10 prompt-injection families, including 150 benign boundary cases paired with related attacks. It is intended to support both attack detection and over-refusal testing in Turkish LLM security workflows.
The new loader:
harm_categorieswhile leaving benign examples uncategorized;The loader is registered with
SeedDatasetProvider, documented in the dataset-loading guide, and linked to a bibliography entry. Attack examples contain adversarial instructions and should only be used in controlled, authorized security testing.Tests and Documentation
uv run pytest tests/unit/datasets/test_turkish_conversation_prompt_injection_dataset.py tests/unit/datasets/test_seed_dataset_provider.py -q: 206 passed.ty, notebook sanitation, documentation structure, and secret detection.jupytext --test-strictfor the updated dataset-loading notebook: passed.AI assistance was used for implementation drafting and test planning. I reviewed the resulting code and documentation, validated them against the live dataset, and ran all checks listed above.